{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 自然语言处理实战 —— 问答系统\n",
    "\n",
    "在NLP（自然语言处理）领域，与我们生活息息相关的就是问答系统（QA），它是机器与人交互最常见的方法。\n",
    "\n",
    "针对实际生活中的各个领域，可以依靠固定领域的数据库来进行特定问题的回答。例如电商智能客服、银行智能回答等，其实质都是问答系统的流程：\n",
    "\n",
    "- 首先对问题内容识别分析\n",
    "\n",
    "- 再基于先验知识，找到合适的回答\n",
    "\n",
    "在前面三期的自然语言处理的实战中，分别体验了命名实体识别、文本分类、文本相似度分析三个 NLP 领域的基础任务。在本期中，将结合命名实体识别和文本相似度这两个任务，即\n",
    "[【华为云 ModelArts-Lab AI实战营】第七期：自然语言处理（I）命名实体识别](https://github.com/huaweicloud/ModelArts-Lab/issues/931)和[【华为云 ModelArts-Lab AI实战营】第九期：自然语言处理（III）文本相似度分析](https://github.com/huaweicloud/ModelArts-Lab/issues/1087)，来完成 NLP 领域的一个上层任务——问答系统（QA）。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 进入ModelArts\n",
    "\n",
    "点击如下链接：https://www.huaweicloud.com/product/modelarts.html ， 进入ModelArts主页。点击“立即使用”按钮，输入用户名和密码登录，进入ModelArts使用页面。\n",
    "\n",
    "### 创建ModelArts notebook\n",
    "\n",
    "下面，我们在ModelArts中创建一个notebook开发环境，ModelArts notebook提供网页版的Python开发环境，可以方便的编写、运行代码，并查看运行结果。\n",
    "\n",
    "第一步：在ModelArts服务主界面依次点击“开发环境”、“创建”\n",
    "\n",
    "![create_nb_create_button](./img/create_nb_create_button.png)\n",
    "\n",
    "第二步：填写notebook所需的参数：\n",
    "\n",
    "| 参数 | 说明 |\n",
    "| - - - - - | - - - - - |\n",
    "| 计费方式 | 按需计费  |\n",
    "| 名称 | Notebook实例名称，如 text_sentiment_analysis |\n",
    "| 工作环境 | Python3 |\n",
    "| 资源池 | 选择\"公共资源池\"即可 |\n",
    "| 类型 | 本案例使用较为复杂的深度神经网络模型，需要较高算力，选择\"GPU\" |\n",
    "| 规格 | 选择\"8核 &#124; 64GiB &#124; 1*p100\" |\n",
    "| 存储配置 | 选择EVS，磁盘规格5GB |\n",
    "\n",
    "第三步：配置好notebook参数后，点击下一步，进入notebook信息预览。确认无误后，点击“立即创建”\n",
    "\n",
    "![create_nb_creation_summary](./img/create_nb_creation_summary.png)\n",
    "\n",
    "第四步：创建完成后，返回开发环境主界面，等待Notebook创建完毕后，打开Notebook，进行下一步操作。\n",
    "![modelarts_notebook_index](./img/modelarts_notebook_index.png)\n",
    "\n",
    "### 在ModelArts中创建开发环境\n",
    "\n",
    "接下来，我们创建一个实际的开发环境，用于后续的实验步骤。\n",
    "\n",
    "第一步：点击下图所示的“打开”按钮，进入刚刚创建的Notebook\n",
    "![inter_dev_env](img/enter_dev_env.png)\n",
    "\n",
    "第二步：创建一个Python3环境的的Notebook。点击右上角的\"New\"，然后创建TensorFlow 1.13.1开发环境。\n",
    "\n",
    "第三步：点击左上方的文件名\"Untitled\"，并输入一个与本实验相关的名称\n",
    "![notebook_untitled_filename](./img/notebook_untitled_filename.png)\n",
    "![notebook_name_the_ipynb](./img/notebook_name_the_ipynb.png)\n",
    "\n",
    "\n",
    "### 在Notebook中编写并执行代码\n",
    "\n",
    "在Notebook中，我们输入一个简单的打印语句，然后点击上方的运行按钮，可以查看语句执行的结果：\n",
    "![run_helloworld](./img/run_helloworld.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 知识库\n",
    "\n",
    "“第74届奥斯卡金像奖的最佳影片是《美丽心灵》”\n",
    "\n",
    "“《鲁迅全集》这本书是1981年出版的”\n",
    "\n",
    "“乔丹出生在美国纽约布鲁克林”。\n",
    "\n",
    "以上就是三条**知识**，而把大量的知识汇聚起来就成为了知识库。我们可以在wiki百科，百度百科等查阅到大量的知识。然而，这些百科的知识组建形式是非结构化的自然语言，这样的组织方式很适合人们阅读但并不适合计算机去处理。为了方便计算机的处理和理解，我们需要更加形式化、简洁化的方式去表示知识——即**三元组（triple）**。知识库通常由大量的三元组组成。\n",
    "\n",
    "“第74届奥斯卡金像奖的最佳影片是《美丽心灵》” 可以用三元组表示为（第74届奥斯卡金像奖, 最佳影片, 美丽心灵）。\n",
    "\n",
    "这里通常把三元组理解为（实体 entity，关系属性 attribute，属性值 value），即“第74届奥斯卡金像奖”为实体, “最佳影片”为关系属性, “美丽心灵”为属性值。\n",
    "\n",
    "进一步的，如果我们把实体看作是结点，把实体关系（包括属性，类别等等）看作是一条边，那么包含了大量三元组的知识库就成为了一个庞大的知识图。如下即为一个小型知识图。\n",
    "\n",
    "![knowledge_graph](./img/knowledge_graph.png)\n",
    "\n",
    "知识图谱又是一个很复杂庞大的领域，近年来也越来越受到人们的重视，在此不进行具体的介绍，感兴趣的同学可以自行学习，并分享到我们的拓展案例或者体验文章中，分享位置是[contrib](https://github.com/huaweicloud/ModelArts-Lab/tree/master/contrib)。\n",
    "\n",
    "\n",
    "## 数据集\n",
    "\n",
    "本实战使用的中文数据集来自NLPCC（自然语言处理与中文计算会议，The Conference on Natural Language Processing and Chinese Computing）2016中的[***Task5：Open Domain Question Answering***](http://tcci.ccf.org.cn/conference/2017/taskdata.php)数据集。数据集包含 14,609 个问答对的训练集和包含 9870 个问答对的测试集。并提供一个知识库，包含 6,502,738 个实体、587,875 个属性以及 43,063,796 个三元组。\n",
    "\n",
    "- 知识库：nlpcc-iccpol-2016.kbqa.kb\n",
    "- 训练集：nlpcc-iccpol-2016.kbqa.traing-data\n",
    "- 测试集：nlpcc-iccpol-2016.kbqa.testing-data\n",
    "\n",
    "知识库文件中每行存储一个事实（fact），即三元组 ( 实体、属性、属性值) 。知识库样例如下所示：\n",
    "\n",
    "```\n",
    "\"希望之星\"英语风采大赛|||中文名|||“希望之星”英语风采大赛\n",
    "\"希望之星\"英语风采大赛|||主办方|||中央电视台科教节目中心\n",
    "\"希望之星\"英语风采大赛|||别名|||\"希望之星\"英语风采大赛\n",
    "\"希望之星\"英语风采大赛|||外文名|||Star of Outlook English Talent Competition\n",
    "\"希望之星\"英语风采大赛|||开始时间|||1998\n",
    "\"希望之星\"英语风采大赛|||比赛形式|||全国选拔\n",
    "\"希望之星\"英语风采大赛|||节目类型|||英语比赛\n",
    "计算机应用基础 ||| 别名 ||| 计算机应用基础\n",
    "计算机应用基础 ||| 中文名 ||| 计算机应用基础\n",
    "计算机应用基础 ||| 作者 ||| 刘晓斌、魏智荣、刘庆生\n",
    "计算机应用基础 ||| 类别 ||| 计算机/网络 > 计算机理论\n",
    "计算机应用基础 ||| 字数 ||| 445000\n",
    "计算机应用基础 ||| ISBN ||| 9787122177513\n",
    "计算机应用基础 ||| 出版社 ||| 化学工业出版社\n",
    "计算机应用基础 ||| 页数 ||| 255\n",
    "计算机应用基础 ||| 开本 ||| 16开\n",
    "计算机应用基础 ||| 出版时间 ||| 2013年\n",
    "计算机应用基础 ||| 装帧 ||| 平装\n",
    "计算机应用基础 ||| 原作者 ||| 刘升贵，黄敏，庄强兵\n",
    "计算机应用基础 ||| 定价 ||| 29.00元\n",
    "```\n",
    "\n",
    "由于原数据集中知识库数据量庞大（2.3G），在本实战中提供的知识库文件仅为节选示例。\n",
    "\n",
    "训练集和测试集为经过[预处理](https://github.com/huangxiangzhou/NLPCC2016KBQA)包含三元组的数据集。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 准备源代码和数据\n",
    "\n",
    "准备案例所需的源代码和数据，相关资源已经保存在OBS中，我们通过ModelArts SDK将资源下载到本地。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully download file modelarts-labs/notebook/DL_nlp_qa/qa.tar.gz from OBS to local ./qa.tar.gz\n",
      "total 375464\r\n",
      "drwxrwxrwx  3 ma-user ma-group      4096 Sep 30 17:31 .\r\n",
      "drwsrwsr-x 20 ma-user ma-group      4096 Sep 30 17:17 ..\r\n",
      "drwxr-x---  2 ma-user ma-group      4096 Sep 30 17:17 .ipynb_checkpoints\r\n",
      "-rw-r-----  1 ma-user ma-group     91607 Sep 30 17:30 qa.ipynb\r\n",
      "-rw-r-----  1 ma-user ma-group 384366229 Sep 30 17:31 qa.tar.gz\r\n"
     ]
    }
   ],
   "source": [
    "from modelarts.session import Session\n",
    "session = Session()\n",
    "\n",
    "if session.region_name == 'cn-north-1':\n",
    "    bucket_path = 'modelarts-labs/notebook/DL_nlp_qa/qa.tar.gz'\n",
    "    \n",
    "elif session.region_name == 'cn-north-4':\n",
    "    bucket_path = 'modelarts-labs-bj4/notebook/DL_nlp_qa/qa.tar.gz'\n",
    "else:\n",
    "    print(\"请更换地区到北京一或北京四\")\n",
    "    \n",
    "session.download_data(bucket_path=bucket_path, path='./qa.tar.gz')\n",
    "\n",
    "!ls -la"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "解压从obs下载的压缩包，解压后删除压缩包。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 108\r\n",
      "drwxrwxrwx  4 ma-user ma-group  4096 Sep 30 17:31 .\r\n",
      "drwsrwsr-x 20 ma-user ma-group  4096 Sep 30 17:17 ..\r\n",
      "drwxr-x---  2 ma-user ma-group  4096 Sep 30 17:17 .ipynb_checkpoints\r\n",
      "drwxr-x---  8 ma-user ma-group  4096 Sep 30 17:15 qa\r\n",
      "-rw-r-----  1 ma-user ma-group 91607 Sep 30 17:30 qa.ipynb\r\n"
     ]
    }
   ],
   "source": [
    "!tar xf ./qa.tar.gz\n",
    "\n",
    "!rm ./qa.tar.gz\n",
    "\n",
    "!ls -la   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "基于知识库的问答（KBQA）拆分为两个主要步骤: **命名实体识别**步骤和**属性映射**步骤。\n",
    "\n",
    "其中，实体识别步骤的目的是找到问句中询问的实体名称，而属性映射步骤的目的在于找到问句中询问的相关属性。\n",
    "\n",
    "下面分别进行这两个步骤。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## 命名实体识别\n",
    "\n",
    "[【华为云 ModelArts-Lab AI实战营】第七期：自然语言处理（I）命名实体识别](https://github.com/huaweicloud/ModelArts-Lab/issues/931)中，进行了中文命名实体识别的任务。\n",
    "\n",
    "在这里，我们同样使用该方法来识别出问句中的实体内容，即知识库的实体，来找到实体对应的知识。两者的区别是，在[实战营的第七期](https://github.com/huaweicloud/ModelArts-Lab/issues/931)中，选择识别的命名实体包括人名、地名、组织机构名，而在本实战中，只需要识别包含在知识库的一个实体。\n",
    "\n",
    "构造NER的数据集，需要根据三元组-Enitity 反向标注问题，给数据集中的Question 打标签。在这里采用BIO的标注方式，将每个元素标注为“B-X”、“I-X”或者“O”：\n",
    "\n",
    "- “B-X”（Begin）表示此元素所在的片段属于X类型并且此元素在此片段的开头；\n",
    "\n",
    "- “I-X”（Inside）表示此元素所在的片段属于X类型并且此元素在此片段的中间位置；\n",
    "\n",
    "- “O”（Outside）表示不属于任何类型。\n",
    "\n",
    "在第七期中，把“X”为“LOC”代表地名，“PER”代表人名，“ORG”代表组织机构名。由于本任务无需区分地点名、人名和组织名，只需要识别出实体，因此用B-LOC, I-LOC代替其他的标注类型，即B-LOC表示实体首字，I-LOC表示实体非首字。\n",
    "\n",
    "使用本实践的数据集进行标注，标注前示例：\n",
    "\n",
    "```\n",
    "《机械设计基础》这本书的作者是谁？     机械设计基础\n",
    "\n",
    "《高等数学》是哪个出版社出版的？        高等数学\n",
    " ```\n",
    " \n",
    "标注后为：\n",
    "\n",
    "![ner_tagging](./img/ner_tagging.png)\n",
    "\n",
    "本实战已为开发者做好命名实体的数据标注工作，，储存在`data/data_ner`文件夹下的`train.txt`中.\n",
    "\n",
    "执行下面程序，查看前50行标注。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "《 O\n",
      "\n",
      "机 B-LOC\n",
      "\n",
      "械 I-LOC\n",
      "\n",
      "设 I-LOC\n",
      "\n",
      "计 I-LOC\n",
      "\n",
      "基 I-LOC\n",
      "\n",
      "础 I-LOC\n",
      "\n",
      "》 O\n",
      "\n",
      "这 O\n",
      "\n",
      "本 O\n",
      "\n",
      "书 O\n",
      "\n",
      "的 O\n",
      "\n",
      "作 O\n",
      "\n",
      "者 O\n",
      "\n",
      "是 O\n",
      "\n",
      "谁 O\n",
      "\n",
      "？ O\n",
      "\n",
      "   \n",
      "\n",
      "《 O\n",
      "\n",
      "高 B-LOC\n",
      "\n",
      "等 I-LOC\n",
      "\n",
      "数 I-LOC\n",
      "\n",
      "学 I-LOC\n",
      "\n",
      "》 O\n",
      "\n",
      "是 O\n",
      "\n",
      "哪 O\n",
      "\n",
      "个 O\n",
      "\n",
      "出 O\n",
      "\n",
      "版 O\n",
      "\n",
      "社 O\n",
      "\n",
      "出 O\n",
      "\n",
      "版 O\n",
      "\n",
      "的 O\n",
      "\n",
      "？ O\n",
      "\n",
      "   \n",
      "\n",
      "《 O\n",
      "\n",
      "线 B-LOC\n",
      "\n",
      "性 I-LOC\n",
      "\n",
      "代 I-LOC\n",
      "\n",
      "数 I-LOC\n",
      "\n",
      "》 O\n",
      "\n",
      "这 O\n",
      "\n",
      "本 O\n",
      "\n",
      "书 O\n",
      "\n",
      "的 O\n",
      "\n",
      "出 O\n",
      "\n",
      "版 O\n",
      "\n",
      "时 O\n",
      "\n",
      "间 O\n",
      "\n",
      "是 O\n",
      "\n"
     ]
    }
   ],
   "source": [
    "with open(\"./qa/data/data_ner/train.txt\", \"r\") as f:\n",
    "    d = f.readlines()\n",
    "    for i in range(50):\n",
    "        print(d[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "本实战中的命名实体识别任务与[实战营的第七期](https://github.com/huaweicloud/ModelArts-Lab/issues/931)步骤基本相同，使用**BERT**模型完成，故不在此列出，封装到`bert_ner.py`文件中。\n",
    "\n",
    "参数设置如下：\n",
    "\n",
    "```\n",
    "max_seq_length = 128          #序列最大长度\n",
    "train_batch_size = 64         #训练批尺寸\n",
    "predict_batch_size = 64        #测试批尺寸\n",
    "learning_rate = 5e-5          #学习率\n",
    "num_train_epochs = 5.0         #训练轮数\n",
    "dropout_rate = 1             #随机失活率\n",
    "warmup_proportion = 0.1        #预热训练\n",
    "```\n",
    "\n",
    "执行下面程序块运行，完成命名实体识别的训练和测试。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Estimator's model_fn (<function model_fn_builder.<locals>.model_fn at 0x7f8dc18b8598>) includes params argument, but params are not passed to Estimator.\n",
      "INFO:tensorflow:Using config: {'_model_dir': './qa/output_ner', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true\n",
      "graph_options {\n",
      "  rewrite_options {\n",
      "    meta_optimizer_iterations: ONE\n",
      "  }\n",
      "}\n",
      ", '_keep_checkpoint_max': 1, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f8dc15c8940>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=1000, num_shards=8, num_cores_per_replica=None, per_host_input_for_training=3, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None), '_cluster': None}\n",
      "INFO:tensorflow:_TPUContext: eval_on_tpu True\n",
      "WARNING:tensorflow:eval_on_tpu ignored because use_tpu is False.\n",
      "INFO:tensorflow:Writing example 0 of 13637\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: train-0\n",
      "INFO:tensorflow:tokens: 《 机 械 设 计 基 础 》 这 本 书 的 作 者 是 谁 ？\n",
      "INFO:tensorflow:input_ids: 101 517 3322 3462 6392 6369 1825 4794 518 6821 3315 741 4638 868 5442 3221 6443 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label_ids: 9 1 6 7 7 7 7 7 1 1 1 1 1 1 1 1 1 1 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: train-1\n",
      "INFO:tensorflow:tokens: 《 高 等 数 学 》 是 哪 个 出 版 社 出 版 的 ？\n",
      "INFO:tensorflow:input_ids: 101 517 7770 5023 3144 2110 518 3221 1525 702 1139 4276 4852 1139 4276 4638 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label_ids: 9 1 6 7 7 7 1 1 1 1 1 1 1 1 1 1 1 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: train-2\n",
      "INFO:tensorflow:tokens: 《 线 性 代 数 》 这 本 书 的 出 版 时 间 是 什 么 ？\n",
      "INFO:tensorflow:input_ids: 101 517 5296 2595 807 3144 518 6821 3315 741 4638 1139 4276 3198 7313 3221 784 720 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label_ids: 9 1 6 7 7 7 1 1 1 1 1 1 1 1 1 1 1 1 1 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: train-3\n",
      "INFO:tensorflow:tokens: 安 德 烈 是 哪 个 国 家 的 人 呢 ？\n",
      "INFO:tensorflow:input_ids: 101 2128 2548 4164 3221 1525 702 1744 2157 4638 782 1450 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label_ids: 9 6 7 7 1 1 1 1 1 1 1 1 1 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: train-4\n",
      "INFO:tensorflow:tokens: 《 线 性 代 数 》 的 i s b n 码 是 什 么 ？\n",
      "INFO:tensorflow:input_ids: 101 517 5296 2595 807 3144 518 4638 151 161 144 156 4772 3221 784 720 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label_ids: 9 1 6 7 7 7 1 1 1 1 1 1 1 1 1 1 1 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "O B-LOC I-LOC I-LOC I-LOC I-LOC I-LOC O O O O O O O O O O\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Writing example 5000 of 13637\n",
      "INFO:tensorflow:Writing example 10000 of 13637\n",
      "INFO:tensorflow:***** Running training *****\n",
      "INFO:tensorflow:  Num examples = 13637\n",
      "INFO:tensorflow:  Batch size = 64\n",
      "INFO:tensorflow:  Num steps = 1065\n",
      "WARNING:tensorflow:From /home/ma-user/anaconda3/envs/TensorFlow-1.13.1/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py:435: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Colocations handled automatically by placer.\n",
      "WARNING:tensorflow:From /home/ma-user/work/qa/bert_ner.py:355: map_and_batch (from tensorflow.contrib.data.python.ops.batching) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.data.experimental.map_and_batch(...)`.\n",
      "WARNING:tensorflow:From /home/ma-user/work/qa/bert_ner.py:342: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Running train on CPU\n",
      "WARNING:tensorflow:From /home/ma-user/work/qa/bert/modeling.py:358: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n",
      "WARNING:tensorflow:From /home/ma-user/work/qa/bert/modeling.py:671: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use keras.layers.dense instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape of input_ids (64, 128)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /home/ma-user/anaconda3/envs/TensorFlow-1.13.1/lib/python3.6/site-packages/tensorflow/contrib/crf/python/ops/crf.py:213: dynamic_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use `keras.layers.RNN(cell)`, which is equivalent to this API\n",
      "WARNING:tensorflow:From /home/ma-user/anaconda3/envs/TensorFlow-1.13.1/lib/python3.6/site-packages/tensorflow/python/training/learning_rate_decay_v2.py:321: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Deprecated in favor of operator or tf.math.divide.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 0 into ./qa/output_ner/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 1.0474\n",
      "INFO:tensorflow:examples/sec: 67.0333\n",
      "INFO:tensorflow:global_step/sec: 1.12179\n",
      "INFO:tensorflow:examples/sec: 71.7943\n",
      "INFO:tensorflow:global_step/sec: 1.12324\n",
      "INFO:tensorflow:examples/sec: 71.8874\n",
      "INFO:tensorflow:global_step/sec: 1.12284\n",
      "INFO:tensorflow:examples/sec: 71.8615\n",
      "INFO:tensorflow:global_step/sec: 1.12351\n",
      "INFO:tensorflow:examples/sec: 71.9046\n",
      "INFO:tensorflow:global_step/sec: 1.12301\n",
      "INFO:tensorflow:examples/sec: 71.8723\n",
      "INFO:tensorflow:Saving checkpoints for 657 into ./qa/output_ner/model.ckpt.\n",
      "WARNING:tensorflow:From /home/ma-user/anaconda3/envs/TensorFlow-1.13.1/lib/python3.6/site-packages/tensorflow/python/training/saver.py:966: remove_checkpoint (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use standard file APIs to delete files with this prefix.\n",
      "INFO:tensorflow:global_step/sec: 1.08961\n",
      "INFO:tensorflow:examples/sec: 69.7349\n",
      "INFO:tensorflow:global_step/sec: 1.12222\n",
      "INFO:tensorflow:examples/sec: 71.8219\n",
      "INFO:tensorflow:global_step/sec: 1.1238\n",
      "INFO:tensorflow:examples/sec: 71.9232\n",
      "INFO:tensorflow:global_step/sec: 1.12346\n",
      "INFO:tensorflow:examples/sec: 71.9013\n",
      "INFO:tensorflow:Saving checkpoints for 1065 into ./qa/output_ner/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 13.384634.\n",
      "INFO:tensorflow:training_loop marked as finished\n",
      "INFO:tensorflow:Writing example 0 of 9016\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-0\n",
      "INFO:tensorflow:tokens: 你 知 道 计 算 机 应 用 基 础 这 本 书 的 作 者 是 谁 吗 ？\n",
      "INFO:tensorflow:input_ids: 101 872 4761 6887 6369 5050 3322 2418 4500 1825 4794 6821 3315 741 4638 868 5442 3221 6443 1408 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label_ids: 9 1 1 1 6 7 7 7 7 7 7 1 1 1 1 1 1 1 1 1 1 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-1\n",
      "INFO:tensorflow:tokens: 计 算 机 应 用 基 础 这 本 书 的 出 版 社 是 那 个 ？\n",
      "INFO:tensorflow:input_ids: 101 6369 5050 3322 2418 4500 1825 4794 6821 3315 741 4638 1139 4276 4852 3221 6929 702 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label_ids: 9 6 7 7 7 7 7 7 1 1 1 1 1 1 1 1 1 1 1 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-2\n",
      "INFO:tensorflow:tokens: 告 诉 我 高 等 数 学 的 出 版 时 间 是 什 么 时 候 ？\n",
      "INFO:tensorflow:input_ids: 101 1440 6401 2769 7770 5023 3144 2110 4638 1139 4276 3198 7313 3221 784 720 3198 952 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label_ids: 9 1 1 1 6 7 7 7 1 1 1 1 1 1 1 1 1 1 1 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-3\n",
      "INFO:tensorflow:tokens: 我 想 知 道 戴 维 斯 是 什 么 国 家 的 人 ？\n",
      "INFO:tensorflow:input_ids: 101 2769 2682 4761 6887 2785 5335 3172 3221 784 720 1744 2157 4638 782 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label_ids: 9 1 1 1 1 6 7 7 1 1 1 1 1 1 1 1 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-4\n",
      "INFO:tensorflow:tokens: 你 知 道 高 等 数 学 的 i s b n 吗 ？\n",
      "INFO:tensorflow:input_ids: 101 872 4761 6887 7770 5023 3144 2110 4638 151 161 144 156 1408 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label_ids: 9 1 1 1 6 7 7 7 1 1 1 1 1 1 1 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "O O O B-LOC I-LOC I-LOC I-LOC I-LOC I-LOC I-LOC O O O O O O O O O O\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Writing example 5000 of 9016\n",
      "INFO:tensorflow:***** Running prediction*****\n",
      "INFO:tensorflow:  Num examples = 9016\n",
      "INFO:tensorflow:  Batch size = 64\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Running infer on CPU\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape of input_ids (?, 128)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "WARNING:tensorflow:From /home/ma-user/anaconda3/envs/TensorFlow-1.13.1/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use standard file APIs to check for files with this prefix.\n",
      "INFO:tensorflow:Restoring parameters from ./qa/output_ner/model.ckpt-1065\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:prediction_loop marked as finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "打印测试评价指标\n",
      "processed 145137 tokens with 9016 phrases; found: 9050 phrases; correct: 8639.\n",
      "accuracy:  99.43%; precision:  95.46%; recall:  95.82%; FB1:  95.64\n",
      "              LOC: precision:  95.46%; recall:  95.82%; FB1:  95.64  9050\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#进行命名实体识别的训练、验证和测试\n",
    "from qa import bert_ner\n",
    "\n",
    "bert_ner.main(_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "至此，问答系统构建的前半部分已经完成，接下来进行后半部分的工作——属性映射。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 属性映射\n",
    "\n",
    "属性映射目的在于找到问句中询问的相关属性，转换成文本相似度问题，即[【华为云 ModelArts-Lab AI实战营】第九期：自然语言处理（III）文本相似度分析](https://github.com/huaweicloud/ModelArts-Lab/issues/1087)。\n",
    "\n",
    "构造用于文本相似度分析的训练集和测试集，构造测试集的整体关系集合，通过提取和去重，获得若干关系集合；每个sample由“问题句+关系属性+label”构成，原始数据中的关系属性的label为 1；从关系集合中随机采样五个属性作为 Negative Samples，label为0。\n",
    "\n",
    "本实战已为开发者做好属性映射的数据处理，储存在`data/data_sim`文件夹下的`train.txt`中。\n",
    "\n",
    "打印训练集前50行的问题、三元组、答案和属性，示例如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "12005\t请问有没有其他出版社出版了东京暗鸦？\t信噪比\t0\n",
      "\n",
      "12006\t请问有没有其他出版社出版了东京暗鸦？\t职位\t0\n",
      "\n",
      "12007\t请问有没有其他出版社出版了东京暗鸦？\t生产年份\t0\n",
      "\n",
      "12008\t请问草根金融论坛的联系方式是什么？\t联络方式\t1\n",
      "\n",
      "12009\t请问草根金融论坛的联系方式是什么？\t设计\t0\n",
      "\n",
      "12010\t请问草根金融论坛的联系方式是什么？\t建置时间\t0\n",
      "\n",
      "12011\t请问草根金融论坛的联系方式是什么？\t原料\t0\n",
      "\n",
      "12012\t请问草根金融论坛的联系方式是什么？\t人口总数\t0\n",
      "\n",
      "12013\t请问草根金融论坛的联系方式是什么？\t产生\t0\n",
      "\n",
      "12014\t请问足跟骨刺是否被纳入了医保体系?\t是否进入医保\t1\n",
      "\n",
      "12015\t请问足跟骨刺是否被纳入了医保体系?\t外观设计\t0\n",
      "\n",
      "12016\t请问足跟骨刺是否被纳入了医保体系?\t初次登场\t0\n",
      "\n",
      "12017\t请问足跟骨刺是否被纳入了医保体系?\t定名人\t0\n",
      "\n",
      "12018\t请问足跟骨刺是否被纳入了医保体系?\tt(厚度)\t0\n",
      "\n",
      "12019\t请问足跟骨刺是否被纳入了医保体系?\t友好城市\t0\n",
      "\n",
      "12020\t请问西北大学的研究生有多少人？\t研究生\t1\n",
      "\n",
      "12021\t请问西北大学的研究生有多少人？\tcpu主频\t0\n",
      "\n",
      "12022\t请问西北大学的研究生有多少人？\t最大输出功率\t0\n",
      "\n",
      "12023\t请问西北大学的研究生有多少人？\t定价\t0\n",
      "\n",
      "12024\t请问西北大学的研究生有多少人？\t注释\t0\n",
      "\n",
      "12025\t请问西北大学的研究生有多少人？\t放大倍率\t0\n",
      "\n",
      "12026\t请问生地饴糖鸡属于什么食物？\t所属类型\t1\n",
      "\n",
      "12027\t请问生地饴糖鸡属于什么食物？\t副职\t0\n",
      "\n",
      "12028\t请问生地饴糖鸡属于什么食物？\tcpu主频\t0\n",
      "\n",
      "12029\t请问生地饴糖鸡属于什么食物？\t青年队\t0\n",
      "\n",
      "12030\t请问生地饴糖鸡属于什么食物？\t主板架构\t0\n",
      "\n",
      "12031\t请问生地饴糖鸡属于什么食物？\t配 音\t0\n",
      "\n",
      "12032\t请问哪个公司出版了《忘忧草》啊？\t音乐公司\t1\n",
      "\n",
      "12033\t请问哪个公司出版了《忘忧草》啊？\t转职业年\t0\n",
      "\n",
      "12034\t请问哪个公司出版了《忘忧草》啊？\t网站性质\t0\n",
      "\n",
      "12035\t请问哪个公司出版了《忘忧草》啊？\t专长\t0\n",
      "\n",
      "12036\t请问哪个公司出版了《忘忧草》啊？\t领导者\t0\n",
      "\n",
      "12037\t请问哪个公司出版了《忘忧草》啊？\t使用平台\t0\n",
      "\n",
      "12038\t请问多花秋海棠属于什么亚组？\t亚组\t1\n",
      "\n",
      "12039\t请问多花秋海棠属于什么亚组？\t产品天线\t0\n",
      "\n",
      "12040\t请问多花秋海棠属于什么亚组？\t人口（2005）\t0\n",
      "\n",
      "12041\t请问多花秋海棠属于什么亚组？\t项目占地\t0\n",
      "\n",
      "12042\t请问多花秋海棠属于什么亚组？\t功能主治\t0\n",
      "\n",
      "12043\t请问多花秋海棠属于什么亚组？\t应用范围\t0\n",
      "\n",
      "12044\t请问广州恒大淘宝足球俱乐部的总经理是谁啊？\t总经理\t1\n",
      "\n",
      "12045\t请问广州恒大淘宝足球俱乐部的总经理是谁啊？\t制作人\t0\n",
      "\n",
      "12046\t请问广州恒大淘宝足球俱乐部的总经理是谁啊？\t属\t0\n",
      "\n",
      "12047\t请问广州恒大淘宝足球俱乐部的总经理是谁啊？\t注册时间\t0\n",
      "\n",
      "12048\t请问广州恒大淘宝足球俱乐部的总经理是谁啊？\t性状\t0\n",
      "\n",
      "12049\t请问广州恒大淘宝足球俱乐部的总经理是谁啊？\t瞄准具型式\t0\n",
      "\n",
      "12050\t请问spark乐驰的驱动方式是什么？\t驱动方式\t1\n",
      "\n",
      "12051\t请问spark乐驰的驱动方式是什么？\t科 室\t0\n",
      "\n",
      "12052\t请问spark乐驰的驱动方式是什么？\t剧本\t0\n",
      "\n",
      "12053\t请问spark乐驰的驱动方式是什么？\t技术人员\t0\n",
      "\n",
      "12054\t请问spark乐驰的驱动方式是什么？\t执行标准\t0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "with open(\"./qa/data/data_sim/train.txt\", \"r\") as f:\n",
    "    d = f.readlines()\n",
    "    for i in range(50):\n",
    "        print(d[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面进行属性映射的训练和测试，与[【华为云 ModelArts-Lab AI实战营】第九期：自然语言处理（III）文本相似度分析](https://github.com/huaweicloud/ModelArts-Lab/issues/1087)步骤基本相同，使用**BERT**模型完成，故不在此列出，封装到`bert_similarity.py`文件中。 \n",
    "\n",
    "参数设置如下：\n",
    "\n",
    "```\n",
    "num_train_epochs = 5        #训练轮数\n",
    "batch_size = 64           #批尺寸\n",
    "learning_rate = 0.00005      #学习率\n",
    "gpu_memory_fraction = 0.9     #gpu使用率\n",
    "layer_indexes = [-2]        #句向量选取\n",
    "max_seq_len = 64          #序列最大长度\n",
    "```\n",
    "\n",
    "执行下面程序块运行，完成属性映射（文本相似度分析）的训练和测试。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using config: {'_model_dir': './qa/output_sim/', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': gpu_options {\n",
      "  per_process_gpu_memory_fraction: 0.9\n",
      "  allow_growth: true\n",
      "}\n",
      ", '_keep_checkpoint_max': 1, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f8dc19d62e8>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n",
      "INFO:tensorflow:Using config: {'_model_dir': './qa/output_sim/', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': gpu_options {\n",
      "  per_process_gpu_memory_fraction: 0.9\n",
      "  allow_growth: true\n",
      "}\n",
      ", '_keep_checkpoint_max': 1, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f8e5f289b70>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n",
      "INFO:tensorflow:Writing example 0 of 75621\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: train-0\n",
      "INFO:tensorflow:tokens: [CLS] 请 问 有 没 有 其 他 出 版 社 出 版 了 东 京 暗 鸦 ？ [SEP] 信 噪 比 [SEP]\n",
      "INFO:tensorflow:input_ids: 101 6435 7309 3300 3766 3300 1071 800 1139 4276 4852 1139 4276 749 691 776 3266 7887 8043 102 928 1692 3683 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: train-1\n",
      "INFO:tensorflow:tokens: [CLS] 请 问 有 没 有 其 他 出 版 社 出 版 了 东 京 暗 鸦 ？ [SEP] 职 位 [SEP]\n",
      "INFO:tensorflow:input_ids: 101 6435 7309 3300 3766 3300 1071 800 1139 4276 4852 1139 4276 749 691 776 3266 7887 8043 102 5466 855 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: train-2\n",
      "INFO:tensorflow:tokens: [CLS] 请 问 有 没 有 其 他 出 版 社 出 版 了 东 京 暗 鸦 ？ [SEP] 生 产 年 份 [SEP]\n",
      "INFO:tensorflow:input_ids: 101 6435 7309 3300 3766 3300 1071 800 1139 4276 4852 1139 4276 749 691 776 3266 7887 8043 102 4495 772 2399 819 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: train-3\n",
      "INFO:tensorflow:tokens: [CLS] 请 问 草 根 金 融 论 坛 的 联 系 方 式 是 什 么 ？ [SEP] 联 络 方 式 [SEP]\n",
      "INFO:tensorflow:input_ids: 101 6435 7309 5770 3418 7032 6084 6389 1781 4638 5468 5143 3175 2466 3221 784 720 8043 102 5468 5317 3175 2466 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 1 (id = 1)\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: train-4\n",
      "INFO:tensorflow:tokens: [CLS] 请 问 草 根 金 融 论 坛 的 联 系 方 式 是 什 么 ？ [SEP] 设 计 [SEP]\n",
      "INFO:tensorflow:input_ids: 101 6435 7309 5770 3418 7032 6084 6389 1781 4638 5468 5143 3175 2466 3221 784 720 8043 102 6392 6369 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n",
      "INFO:tensorflow:Writing example 10000 of 75621\n",
      "INFO:tensorflow:Writing example 20000 of 75621\n",
      "INFO:tensorflow:Writing example 30000 of 75621\n",
      "INFO:tensorflow:Writing example 40000 of 75621\n",
      "INFO:tensorflow:Writing example 50000 of 75621\n",
      "INFO:tensorflow:Writing example 60000 of 75621\n",
      "INFO:tensorflow:Writing example 70000 of 75621\n",
      "INFO:tensorflow:***** Running training *****\n",
      "INFO:tensorflow:  Num examples = 75621\n",
      "INFO:tensorflow:  Batch size = 64\n",
      "INFO:tensorflow:  Num steps = 5907\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 0 into ./qa/output_sim/model.ckpt.\n",
      "INFO:tensorflow:loss = 0.80406123, step = 0\n",
      "INFO:tensorflow:global_step/sec: 1.56336\n",
      "INFO:tensorflow:loss = 0.12109105, step = 100 (63.966 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.17991\n",
      "INFO:tensorflow:loss = 0.014379987, step = 200 (45.873 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18049\n",
      "INFO:tensorflow:loss = 0.037250243, step = 300 (45.861 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.17971\n",
      "INFO:tensorflow:loss = 0.004695637, step = 400 (45.878 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.17942\n",
      "INFO:tensorflow:loss = 0.0020469145, step = 500 (45.884 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.17993\n",
      "INFO:tensorflow:loss = 0.0147804385, step = 600 (45.873 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.17918\n",
      "INFO:tensorflow:loss = 0.00692276, step = 700 (45.889 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18035\n",
      "INFO:tensorflow:loss = 0.04618354, step = 800 (45.864 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18008\n",
      "INFO:tensorflow:loss = 0.019808413, step = 900 (45.870 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.1802\n",
      "INFO:tensorflow:loss = 0.0062913853, step = 1000 (45.867 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18033\n",
      "INFO:tensorflow:loss = 0.0133275, step = 1100 (45.864 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.17894\n",
      "INFO:tensorflow:loss = 0.037112594, step = 1200 (45.894 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 1253 into ./qa/output_sim/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 1.72351\n",
      "INFO:tensorflow:loss = 0.12576269, step = 1300 (58.021 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.17812\n",
      "INFO:tensorflow:loss = 0.07432252, step = 1400 (45.911 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.17993\n",
      "INFO:tensorflow:loss = 0.18995932, step = 1500 (45.873 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.17927\n",
      "INFO:tensorflow:loss = 0.0027036057, step = 1600 (45.887 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18044\n",
      "INFO:tensorflow:loss = 0.0038441897, step = 1700 (45.862 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.181\n",
      "INFO:tensorflow:loss = 0.014303254, step = 1800 (45.851 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.1825\n",
      "INFO:tensorflow:loss = 0.00049416925, step = 1900 (45.819 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18204\n",
      "INFO:tensorflow:loss = 0.056986697, step = 2000 (45.829 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.1825\n",
      "INFO:tensorflow:loss = 0.11168129, step = 2100 (45.819 sec)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:global_step/sec: 2.18275\n",
      "INFO:tensorflow:loss = 0.00057690404, step = 2200 (45.814 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18302\n",
      "INFO:tensorflow:loss = 0.00024280635, step = 2300 (45.808 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18192\n",
      "INFO:tensorflow:loss = 0.0028881684, step = 2400 (45.831 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18054\n",
      "INFO:tensorflow:loss = 0.006113374, step = 2500 (45.860 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 2536 into ./qa/output_sim/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 1.68529\n",
      "INFO:tensorflow:loss = 0.0013344908, step = 2600 (59.337 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18049\n",
      "INFO:tensorflow:loss = 0.0038962504, step = 2700 (45.861 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18183\n",
      "INFO:tensorflow:loss = 0.006715819, step = 2800 (45.834 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.17844\n",
      "INFO:tensorflow:loss = 0.055857826, step = 2900 (45.904 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.1794\n",
      "INFO:tensorflow:loss = 0.00040053533, step = 3000 (45.884 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18079\n",
      "INFO:tensorflow:loss = 0.0020485804, step = 3100 (45.855 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18134\n",
      "INFO:tensorflow:loss = 0.00025651336, step = 3200 (45.844 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18023\n",
      "INFO:tensorflow:loss = 0.00030929683, step = 3300 (45.867 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18217\n",
      "INFO:tensorflow:loss = 0.0010573445, step = 3400 (45.827 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.1811\n",
      "INFO:tensorflow:loss = 0.00039029078, step = 3500 (45.848 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.1793\n",
      "INFO:tensorflow:loss = 0.0008383829, step = 3600 (45.886 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.17883\n",
      "INFO:tensorflow:loss = 0.007717145, step = 3700 (45.896 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18169\n",
      "INFO:tensorflow:loss = 0.00049783796, step = 3800 (45.836 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 3815 into ./qa/output_sim/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 1.72628\n",
      "INFO:tensorflow:loss = 0.0031695478, step = 3900 (57.928 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18009\n",
      "INFO:tensorflow:loss = 0.00032910623, step = 4000 (45.869 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18109\n",
      "INFO:tensorflow:loss = 0.00077105424, step = 4100 (45.849 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18079\n",
      "INFO:tensorflow:loss = 0.00023945066, step = 4200 (45.855 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18095\n",
      "INFO:tensorflow:loss = 8.450498e-05, step = 4300 (45.852 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.17976\n",
      "INFO:tensorflow:loss = 0.0001537561, step = 4400 (45.876 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.17887\n",
      "INFO:tensorflow:loss = 0.00019694449, step = 4500 (45.895 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18123\n",
      "INFO:tensorflow:loss = 0.0006497489, step = 4600 (45.846 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18059\n",
      "INFO:tensorflow:loss = 0.00040960687, step = 4700 (45.859 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.17921\n",
      "INFO:tensorflow:loss = 0.00014905719, step = 4800 (45.888 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.1773\n",
      "INFO:tensorflow:loss = 0.00028763563, step = 4900 (45.928 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18012\n",
      "INFO:tensorflow:loss = 0.00081312575, step = 5000 (45.869 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 5097 into ./qa/output_sim/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 1.71021\n",
      "INFO:tensorflow:loss = 0.000552332, step = 5100 (58.472 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18294\n",
      "INFO:tensorflow:loss = 0.00016604707, step = 5200 (45.810 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18256\n",
      "INFO:tensorflow:loss = 0.00019858233, step = 5300 (45.818 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.1825\n",
      "INFO:tensorflow:loss = 7.9740086e-05, step = 5400 (45.819 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18086\n",
      "INFO:tensorflow:loss = 0.0010153695, step = 5500 (45.853 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18148\n",
      "INFO:tensorflow:loss = 0.00014651103, step = 5600 (45.840 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18126\n",
      "INFO:tensorflow:loss = 7.378128e-05, step = 5700 (45.845 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18139\n",
      "INFO:tensorflow:loss = 0.0005076176, step = 5800 (45.842 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18048\n",
      "INFO:tensorflow:loss = 9.0745336e-05, step = 5900 (45.862 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 5907 into ./qa/output_sim/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 7.0486276e-05.\n",
      "INFO:tensorflow:Using config: {'_model_dir': './qa/output_sim/', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': gpu_options {\n",
      "  per_process_gpu_memory_fraction: 0.9\n",
      "  allow_growth: true\n",
      "}\n",
      ", '_keep_checkpoint_max': 1, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f8dba63ca90>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n",
      "INFO:tensorflow:Writing example 0 of 12005\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-0\n",
      "INFO:tensorflow:tokens: [CLS] 《 机 械 设 计 基 础 》 这 本 书 的 作 者 是 谁 ？ [SEP] 作 者 [SEP]\n",
      "INFO:tensorflow:input_ids: 101 517 3322 3462 6392 6369 1825 4794 518 6821 3315 741 4638 868 5442 3221 6443 8043 102 868 5442 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 1 (id = 1)\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-1\n",
      "INFO:tensorflow:tokens: [CLS] 《 机 械 设 计 基 础 》 这 本 书 的 作 者 是 谁 ？ [SEP] 各 方 兵 力 [SEP]\n",
      "INFO:tensorflow:input_ids: 101 517 3322 3462 6392 6369 1825 4794 518 6821 3315 741 4638 868 5442 3221 6443 8043 102 1392 3175 1070 1213 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-2\n",
      "INFO:tensorflow:tokens: [CLS] 《 机 械 设 计 基 础 》 这 本 书 的 作 者 是 谁 ？ [SEP] 地 区 * * [SEP]\n",
      "INFO:tensorflow:input_ids: 101 517 3322 3462 6392 6369 1825 4794 518 6821 3315 741 4638 868 5442 3221 6443 8043 102 1765 1277 115 115 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-3\n",
      "INFO:tensorflow:tokens: [CLS] 《 机 械 设 计 基 础 》 这 本 书 的 作 者 是 谁 ？ [SEP] 音 乐 成 就 [SEP]\n",
      "INFO:tensorflow:input_ids: 101 517 3322 3462 6392 6369 1825 4794 518 6821 3315 741 4638 868 5442 3221 6443 8043 102 7509 727 2768 2218 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-4\n",
      "INFO:tensorflow:tokens: [CLS] 《 机 械 设 计 基 础 》 这 本 书 的 作 者 是 谁 ？ [SEP] 所 有 权 者 [SEP]\n",
      "INFO:tensorflow:input_ids: 101 517 3322 3462 6392 6369 1825 4794 518 6821 3315 741 4638 868 5442 3221 6443 8043 102 2792 3300 3326 5442 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n",
      "INFO:tensorflow:Writing example 10000 of 12005\n",
      "INFO:tensorflow:***** Running evaluation *****\n",
      "INFO:tensorflow:  Num examples = 12005\n",
      "INFO:tensorflow:  Batch size = 64\n",
      "INFO:tensorflow:Using config: {'_model_dir': './qa/output_sim/', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': gpu_options {\n",
      "  per_process_gpu_memory_fraction: 0.9\n",
      "  allow_growth: true\n",
      "}\n",
      ", '_keep_checkpoint_max': 1, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f8d9d56ada0>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "WARNING:tensorflow:From /home/ma-user/anaconda3/envs/TensorFlow-1.13.1/lib/python3.6/site-packages/tensorflow/python/ops/metrics_impl.py:455: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2019-09-30T10:37:04Z\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from ./qa/output_sim/model.ckpt-5907\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Finished evaluation at 2019-09-30-10:37:34\n",
      "INFO:tensorflow:Saving dict for global step 5907: eval_accuracy = 0.9876718, eval_auc = 0.9706281, eval_loss = 0.0823417, global_step = 5907, loss = 0.08273375\n",
      "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 5907: ./qa/output_sim/model.ckpt-5907\n",
      "INFO:tensorflow:***** Eval results *****\n",
      "INFO:tensorflow:  eval_accuracy = 0.9876718\n",
      "INFO:tensorflow:  eval_auc = 0.9706281\n",
      "INFO:tensorflow:  eval_loss = 0.0823417\n",
      "INFO:tensorflow:  global_step = 5907\n",
      "INFO:tensorflow:  loss = 0.08273375\n"
     ]
    }
   ],
   "source": [
    "from qa import bert_similarity \n",
    "import tensorflow as tf\n",
    "\n",
    "#BertSim为BERT相似度检验类\n",
    "sim = bert_similarity.BertSim()\n",
    "\n",
    "#在训练集上进行相似度模型训练\n",
    "sim.set_mode(tf.estimator.ModeKeys.TRAIN)\n",
    "sim.train()\n",
    "\n",
    "#在验证集上进行相似度验证\n",
    "sim.set_mode(tf.estimator.ModeKeys.EVAL)\n",
    "sim.eval()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来，整合前面两个步骤，进行完整的基于知识库的问答系统构建。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 问答系统（QA）\n",
    "\n",
    "整合以上两个步骤，就可以完成一个简单的基于知识库的问答系统。下面为具体说明：\n",
    "\n",
    "####  1. 命名实体识别：输入问题，使用BERT模型得到问题中的实体，在知识库中检索出包含该实体的所有知识组合。\n",
    "\n",
    "#### 2. 属性映射：在包含实体的知识组合中，进行文本相似度分析寻找答案，又可分为非语义匹配和语义匹配。\n",
    "\n",
    "  - **非语义匹配**：如果一个知识三元组的关系属性是输入问题的子集（相当于字符串匹配），则该三元组对应的答案匹配为正确答案。非语义匹配步骤可以大大加速匹配。\n",
    "\n",
    "  - **语义匹配**：即可转化为分类问题，利用BERT模型计算输入问题与知识三元组的相似度，将最相近的三元组对应的答案匹配为正确答案。\n",
    "\n",
    "原数据集中知识库数据量庞大（2.3G），共43,063,796个三元组，在本实战中选择使用训练集的14620个问答对对应的三元组生成知识库，用于完成简易问答任务。若需要使用完整知识库，请自行下载[Task 5: Open Domain Question Answering](http://tcci.ccf.org.cn/conference/2017/taskdata.php)使用。（由于原知识库较大，建议使用数据库进行存储、查找等操作。）\n",
    "\n",
    "本实战已为开发者做好三元组知识库的数据处理，储存在`data/data_qa`文件夹下的`triple.txt`中。\n",
    "\n",
    "执行下面程序，读取三元组知识库，储存在`triple_data`中，并打印前50条知识。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "          0     1                                   2\n",
      "0    机械设计基础    作者                         杨可桢，程光蕴，李仲生\n",
      "1      高等数学   出版社                             武汉大学出版社\n",
      "2      线性代数  出版时间                          2013/12/30\n",
      "3       安德烈    国籍                                 摩纳哥\n",
      "4      线性代数  isbn                   978-7-111-36843-4\n",
      "5      高等数学    书名                          高等数学一（微积分）\n",
      "6      万达广场   外文名                    amoy wanda plaza\n",
      "7        李明  出生日期                              1963.1\n",
      "8     韩娱守护力  小说进度                                  连载\n",
      "9        夏想  连载网站                                潇湘书院\n",
      "10  大学计算机基础    页数                                 272\n",
      "11      城关镇   中文名                            城关镇[临澧县]\n",
      "12       李明   出生地                                青海湟源\n",
      "13  大学计算机基础    定价                                25 元\n",
      "14       李军    职业                      河南省林业厅党组成员、副厅长\n",
      "15       文庙  中文名称                                  文庙\n",
      "16       杨明    民族                                  苗族\n",
      "17      毛泽东    装帧                              平装(无盘)\n",
      "18      西游记    类型                                  奇幻\n",
      "19       王娟    性别                                   女\n",
      "20  计算机应用基础    开本                                  16\n",
      "21       周迅  公司名称                      安徽龙润农业科技开发有限公司\n",
      "22     博士来拜    类别                               油画，壁画\n",
      "23       李军  毕业院校                           天津卫生职工医学院\n",
      "24       陈平    国籍                                  中国\n",
      "25     万达广场  总部地点  重庆市南岸区江南大道8号（重庆万达）/重庆市万州区北滨路（万州万达）\n",
      "26       张宇    性别                                   女\n",
      "27      城关镇    面积                          134.27平方公里\n",
      "28      城关镇  所属地区                                中国西南\n",
      "29      和平村  地理位置                           山东省微山县南阳镇\n",
      "30     西城社区  成立时间                             2006年6月\n",
      "31       王辉    职业                        广东省广播电影电视局官员\n",
      "32     东创建国  经营范围                      整车销售、售后服务、配件供应\n",
      "33       茴香  中文学名                                  茴香\n",
      "34       王军  出生年月                             1962年7月\n",
      "35      丹顶鹤     界                                 动物界\n",
      "36       王平    别名                         原名王惟允，曾用名王明\n",
      "37       李军  主要成就                   中国美术学会会员 中国陶瓷协会会员\n",
      "38      tcl  公司性质                                  私企\n",
      "39      城关镇    人口                                  5万\n",
      "40    圆锥花序组     科                  石竹科caryophyllaceae\n",
      "41     匆匆那年   英文名                        back in time\n",
      "42     《兄弟》    类别                                 图书 \n",
      "43       刘勇  代表作品                     《中国现代作家的宗教文化情结》\n",
      "44    多花秋海棠     门                  被子植物门 angiospermae\n",
      "45    类胡萝卜素    性质                                  色素\n",
      "46       王军    籍贯                                甘肃高台\n",
      "47    命运石之门  游戏类型                             假想科学adv\n",
      "48    圆锥花序组     目                  中央种子目centrospermae\n",
      "49      凤凰山    地点                              山西省定襄县\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "triple_data = pd.read_csv('./qa/data/data_qa/triple.txt', encoding='utf-8', sep='\\t', header=None)\n",
    "print(triple_data[:50])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "建立好知识库后，即可进行基于知识库的问答系统搭建，问答步骤依次为：\n",
    "\n",
    "1. 请提问：输入与知识库相关问题\n",
    "\n",
    "- 打印问题token，实体标注结果，并输出识别实体\n",
    "\n",
    "- 找到结果的可能来源知识集合并打印\n",
    "\n",
    "- 属性映射：分别进行语义匹配（查找属性是否在问题中）和非语义匹配（相似度匹配）\n",
    "\n",
    "- 相似度最高即为匹配答案，并打印答案来源知识及用时\n",
    "\n",
    "- 可进行多轮问答，输入回车结束在线KBQA\n",
    "\n",
    "#### 首先导入依赖包"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using config: {'_model_dir': './qa/output_sim/', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': gpu_options {\n",
      "  per_process_gpu_memory_fraction: 0.9\n",
      "  allow_growth: true\n",
      "}\n",
      ", '_keep_checkpoint_max': 1, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f8da77a8b38>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n",
      "WARNING:tensorflow:From /home/ma-user/anaconda3/envs/TensorFlow-1.13.1/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py:429: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "tf.py_func is deprecated in TF V2. Instead, use\n",
      "    tf.py_function, which takes a python function which manipulates tf eager\n",
      "    tensors instead of numpy arrays. It's easy to convert a tf eager tensor to\n",
      "    an ndarray (just call tensor.numpy()) but having access to eager tensors\n",
      "    means `tf.py_function`s can use accelerators such as GPUs as well as\n",
      "    being differentiable using a gradient tape.\n",
      "    \n",
      "INFO:tensorflow:Calling model_fn.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "checkpoint path:./qa/output_ner/checkpoint\n",
      "going to restore checkpoint\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./qa/output_ner/model.ckpt-1065\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from qa.bert import tokenization\n",
    "from qa.bert_similarity import BertSim\n",
    "from qa.bert_qa import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 读取BERT预训练模型中文字典"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Done calling model_fn.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['今', '天', '的', '天', '气', '真', '好', '！']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer = tokenization.FullTokenizer(vocab_file=vocab_file, do_lower_case=True)\n",
    "\n",
    "tokenizer.tokenize(\"今天的天气真好！\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 分别获得每个字的字向量、位置向量、文本向量和标签"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def convert(line):\n",
    "    feature = convert_single_example(0, line, label_list, max_seq_length, tokenizer, 'p')\n",
    "    input_ids = np.reshape([feature.input_ids],(batch_size, max_seq_length))\n",
    "    input_mask = np.reshape([feature.input_mask],(batch_size, max_seq_length))\n",
    "    segment_ids = np.reshape([feature.segment_ids],(batch_size, max_seq_length))\n",
    "    label_ids =np.reshape([feature.label_ids],(batch_size, max_seq_length))\n",
    "    return input_ids, input_mask, segment_ids, label_ids"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 构建问答系统"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def kbqa():\n",
    "    while True:\n",
    "        print('\\n\\033[1;31m请提问:\\033[0m\\n')\n",
    "        sentence = str(input())\n",
    "        if len(sentence) == 0:\n",
    "            print(\"再见啦！\")\n",
    "            return\n",
    "        \n",
    "        sentence_ = tokenizer.tokenize(sentence)\n",
    "        print('\\n你的问题是:{}'.format(sentence_))\n",
    "        input_ids, input_mask, segment_ids, label_ids = convert(sentence_)\n",
    "\n",
    "        feed_dict = {input_ids_p: input_ids,\n",
    "                     input_mask_p: input_mask,\n",
    "                     segment_ids_p:segment_ids,\n",
    "                     label_ids_p:label_ids}\n",
    "\n",
    "        pred_ids_result = sess.run([pred_ids], feed_dict)\n",
    "        pred_label_result = convert_id_to_label(pred_ids_result, id2label)\n",
    "        print(\"\\n实体标注结果为：\",pred_label_result)\n",
    "        result = strage_combined_link_org_loc(sentence_, pred_label_result[0])\n",
    "        print('\\n识别的实体是：{}'.format(''.join(result)),'\\n')\n",
    "\n",
    "        ans_range = []\n",
    "        for j in range(len(triple_data)):\n",
    "            if triple_data[0][j] == result[0]:\n",
    "                triple_range = [triple_data[0][j],triple_data[1][j],triple_data[2][j]]\n",
    "                print(\"结果可能来自：\",triple_range)\n",
    "                ans_range.append(triple_range)\n",
    "                \n",
    "        ans = None\n",
    "        ans_base = None\n",
    "        score = 0\n",
    "\n",
    "        for k in range(len(ans_range)):\n",
    "            print(\"\\n知识三元组%d：\"%(k+1),ans_range[k][0],ans_range[k][1],ans_range[k][2])\n",
    "\n",
    "            #非语义匹配\n",
    "            if ans_range[k][1] in sentence:\n",
    "                print(\"属性“\",ans_range[k][1],\"”在问题中\")\n",
    "                ans_ = 1\n",
    "\n",
    "            #语义匹配\n",
    "            else:                \n",
    "                ans_ = max(bs.predict(ans_range[k][0]+ans_range[k][1]+ans_range[k][2],sentence)[0])\n",
    "            print(\"相似度为：\",ans_)\n",
    "\n",
    "            if score < ans_:\n",
    "                score = ans_\n",
    "                ans = ans_range[k][2]\n",
    "                ans_base = ans_range[k]\n",
    "\n",
    "        if score < 0.8:\n",
    "            print(\"\\n\\033[1;31m答案不确定\\033[0m\")\n",
    "            \n",
    "        else:\n",
    "            print(\"\\n答案来自三元组：\",ans_base[0],ans_base[1],ans_base[2])\n",
    "            print(\"相似度为：\", score)\n",
    "            print(\"\\n\\033[1;31m答案是：{}\\033[0m\".format(ans))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 下面就可以运行`kbqa()`来进行基于知识库的多轮在线问答。\n",
    "\n",
    "### 注意，由于使用的知识库规格限制，请先大致浏览`./qa/data/data_qa/`文件夹下`triple.txt`文件中的三元组知识，以免无关问题造成问答效果不佳。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\u001b[1;31m请提问:\u001b[0m\n",
      "\n",
      "西游记每集多长时间？\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-0\n",
      "INFO:tensorflow:tokens: [CLS] 西 游 记 类 型 奇 幻 [SEP] 西 游 记 每 集 多 长 时 间 ？ [SEP]\n",
      "INFO:tensorflow:input_ids: 101 6205 3952 6381 5102 1798 1936 2404 102 6205 3952 6381 3680 7415 1914 7270 3198 7313 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-0\n",
      "INFO:tensorflow:tokens: [CLS] 西 游 记 首 播 时 间 2010 年 1 月 3 日 [SEP] 西 游 记 每 集 多 长 时 间 ？ [SEP]\n",
      "INFO:tensorflow:input_ids: 101 6205 3952 6381 7674 3064 3198 7313 8166 2399 122 3299 124 3189 102 6205 3952 6381 3680 7415 1914 7270 3198 7313 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "你的问题是:['西', '游', '记', '每', '集', '多', '长', '时', '间', '？']\n",
      "\n",
      "实体标注结果为： [['B-LOC', 'I-LOC', 'I-LOC', 'O', 'O', 'O', 'O', 'O', 'O', 'O']]\n",
      "\n",
      "识别的实体是：西游记 \n",
      "\n",
      "结果可能来自： ['西游记', '类型', '奇幻']\n",
      "结果可能来自： ['西游记', '首播时间', '2010年1月3日']\n",
      "结果可能来自： ['西游记', '国家／地区', '中国大陆']\n",
      "结果可能来自： ['西游记', '每集长度', '54分钟']\n",
      "结果可能来自： ['西游记', '播映', '中央电视台']\n",
      "\n",
      "知识三元组1： 西游记 类型 奇幻\n",
      "相似度为： 0.97977567\n",
      "\n",
      "知识三元组2： 西游记 首播时间 2010年1月3日\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-0\n",
      "INFO:tensorflow:tokens: [CLS] 西 游 记 国 家 ／ 地 区 中 国 大 陆 [SEP] 西 游 记 每 集 多 长 时 间 ？ [SEP]\n",
      "INFO:tensorflow:input_ids: 101 6205 3952 6381 1744 2157 8027 1765 1277 704 1744 1920 7355 102 6205 3952 6381 3680 7415 1914 7270 3198 7313 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-0\n",
      "INFO:tensorflow:tokens: [CLS] 西 游 记 每 集 长 度 54 分 钟 [SEP] 西 游 记 每 集 多 长 时 间 ？ [SEP]\n",
      "INFO:tensorflow:input_ids: 101 6205 3952 6381 3680 7415 7270 2428 8267 1146 7164 102 6205 3952 6381 3680 7415 1914 7270 3198 7313 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-0\n",
      "INFO:tensorflow:tokens: [CLS] 西 游 记 播 映 中 央 电 视 台 [SEP] 西 游 记 每 集 多 长 时 间 ？ [SEP]\n",
      "INFO:tensorflow:input_ids: 101 6205 3952 6381 3064 3216 704 1925 4510 6228 1378 102 6205 3952 6381 3680 7415 1914 7270 3198 7313 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "相似度为： 0.9963476\n",
      "\n",
      "知识三元组3： 西游记 国家／地区 中国大陆\n",
      "相似度为： 0.675162\n",
      "\n",
      "知识三元组4： 西游记 每集长度 54分钟\n",
      "相似度为： 0.9998462\n",
      "\n",
      "知识三元组5： 西游记 播映 中央电视台\n",
      "相似度为： 0.99562705\n",
      "\n",
      "答案来自三元组： 西游记 每集长度 54分钟\n",
      "相似度为： 0.9998462\n",
      "\n",
      "\u001b[1;31m答案是：54分钟\u001b[0m\n",
      "\n",
      "\u001b[1;31m请提问:\u001b[0m\n",
      "\n",
      "线性代数的出版时间是什么时候？\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-0\n",
      "INFO:tensorflow:tokens: [CLS] 线 性 代 数 isbn ##97 ##8 - 7 - 111 - 368 ##43 - 4 [SEP] 线 性 代 数 的 出 版 时 间 是 什 么 时 候 ？ [SEP]\n",
      "INFO:tensorflow:input_ids: 101 5296 2595 807 3144 8446 9410 8156 118 128 118 8932 118 11642 9433 118 125 102 5296 2595 807 3144 4638 1139 4276 3198 7313 3221 784 720 3198 952 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-0\n",
      "INFO:tensorflow:tokens: [CLS] 线 性 代 数 书 号 368 ##43 [SEP] 线 性 代 数 的 出 版 时 间 是 什 么 时 候 ？ [SEP]\n",
      "INFO:tensorflow:input_ids: 101 5296 2595 807 3144 741 1384 11642 9433 102 5296 2595 807 3144 4638 1139 4276 3198 7313 3221 784 720 3198 952 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "你的问题是:['线', '性', '代', '数', '的', '出', '版', '时', '间', '是', '什', '么', '时', '候', '？']\n",
      "\n",
      "实体标注结果为： [['B-LOC', 'I-LOC', 'I-LOC', 'I-LOC', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O']]\n",
      "\n",
      "识别的实体是：线性代数 \n",
      "\n",
      "结果可能来自： ['线性代数', '出版时间', '2013/12/30']\n",
      "结果可能来自： ['线性代数', 'isbn', '978-7-111-36843-4']\n",
      "结果可能来自： ['线性代数', '书号', '36843']\n",
      "结果可能来自： ['线性代数', '作者', '侯亚君']\n",
      "结果可能来自： ['线性代数', '出版社', '清华大学出版社']\n",
      "结果可能来自： ['线性代数', '定价', '26元']\n",
      "\n",
      "知识三元组1： 线性代数 出版时间 2013/12/30\n",
      "属性“ 出版时间 ”在问题中\n",
      "相似度为： 1\n",
      "\n",
      "知识三元组2： 线性代数 isbn 978-7-111-36843-4\n",
      "相似度为： 0.9995728\n",
      "\n",
      "知识三元组3： 线性代数 书号 36843\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-0\n",
      "INFO:tensorflow:tokens: [CLS] 线 性 代 数 作 者 侯 亚 君 [SEP] 线 性 代 数 的 出 版 时 间 是 什 么 时 候 ？ [SEP]\n",
      "INFO:tensorflow:input_ids: 101 5296 2595 807 3144 868 5442 908 762 1409 102 5296 2595 807 3144 4638 1139 4276 3198 7313 3221 784 720 3198 952 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-0\n",
      "INFO:tensorflow:tokens: [CLS] 线 性 代 数 出 版 社 清 华 大 学 出 版 社 [SEP] 线 性 代 数 的 出 版 时 间 是 什 么 时 候 ？ [SEP]\n",
      "INFO:tensorflow:input_ids: 101 5296 2595 807 3144 1139 4276 4852 3926 1290 1920 2110 1139 4276 4852 102 5296 2595 807 3144 4638 1139 4276 3198 7313 3221 784 720 3198 952 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-0\n",
      "INFO:tensorflow:tokens: [CLS] 线 性 代 数 定 价 26 元 [SEP] 线 性 代 数 的 出 版 时 间 是 什 么 时 候 ？ [SEP]\n",
      "INFO:tensorflow:input_ids: 101 5296 2595 807 3144 2137 817 8153 1039 102 5296 2595 807 3144 4638 1139 4276 3198 7313 3221 784 720 3198 952 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "相似度为： 0.9996922\n",
      "\n",
      "知识三元组4： 线性代数 作者 侯亚君\n",
      "相似度为： 0.97082436\n",
      "\n",
      "知识三元组5： 线性代数 出版社 清华大学出版社\n",
      "相似度为： 0.9994628\n",
      "\n",
      "知识三元组6： 线性代数 定价 26元\n",
      "相似度为： 0.98183894\n",
      "\n",
      "答案来自三元组： 线性代数 出版时间 2013/12/30\n",
      "相似度为： 1\n",
      "\n",
      "\u001b[1;31m答案是：2013/12/30\u001b[0m\n",
      "\n",
      "\u001b[1;31m请提问:\u001b[0m\n",
      "\n",
      "迈克尔·杰克逊的资产是多少？\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-0\n",
      "INFO:tensorflow:tokens: [CLS] 迈 克 尔 · 杰 克 逊 逝 世 2009 年 6 月 25 日 （ 50 岁 ） 美 国 加 利 福 尼 亚 州 洛 杉 矶 市 荷 尔 贝 山 [SEP] 迈 克 尔 · 杰 克 逊 的 资 产 是 多 少 ？ [SEP]\n",
      "INFO:tensorflow:input_ids: 101 6815 1046 2209 185 3345 1046 6849 6860 686 8170 2399 127 3299 8132 3189 8020 8145 2259 8021 5401 1744 1217 1164 4886 2225 762 2336 3821 3329 4768 2356 5792 2209 6564 2255 102 6815 1046 2209 185 3345 1046 6849 4638 6598 772 3221 1914 2208 8043 102 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "你的问题是:['迈', '克', '尔', '·', '杰', '克', '逊', '的', '资', '产', '是', '多', '少', '？']\n",
      "\n",
      "实体标注结果为： [['B-LOC', 'I-LOC', 'I-LOC', 'I-LOC', 'I-LOC', 'I-LOC', 'I-LOC', 'O', 'O', 'O', 'O', 'O', 'O', 'O']]\n",
      "\n",
      "识别的实体是：迈克尔·杰克逊 \n",
      "\n",
      "结果可能来自： ['迈克尔·杰克逊', '逝世', '2009年6月25日（50岁） 美国加利福尼亚州洛杉矶市荷尔贝山']\n",
      "结果可能来自： ['迈克尔·杰克逊', '厂牌', '钢城唱片 摩城唱片 史诗唱片 遗产唱片 mjj制作']\n",
      "结果可能来自： ['迈克尔·杰克逊', '净资产', '▲ 11.78亿美元 （2009年估算）[1]']\n",
      "\n",
      "知识三元组1： 迈克尔·杰克逊 逝世 2009年6月25日（50岁） 美国加利福尼亚州洛杉矶市荷尔贝山\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-0\n",
      "INFO:tensorflow:tokens: [CLS] 迈 克 尔 · 杰 克 逊 厂 牌 钢 城 唱 片 摩 城 唱 片 史 诗 唱 片 遗 产 唱 片 m ##j ##j 制 作 [SEP] 迈 克 尔 · 杰 克 逊 的 资 产 是 多 少 ？ [SEP]\n",
      "INFO:tensorflow:input_ids: 101 6815 1046 2209 185 3345 1046 6849 1322 4277 7167 1814 1548 4275 3040 1814 1548 4275 1380 6408 1548 4275 6890 772 1548 4275 155 8334 8334 1169 868 102 6815 1046 2209 185 3345 1046 6849 4638 6598 772 3221 1914 2208 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-0\n",
      "INFO:tensorflow:tokens: [CLS] 迈 克 尔 · 杰 克 逊 净 资 产 ▲ 11 . 78 亿 美 元 （ 2009 年 估 算 ） [ 1 ] [SEP] 迈 克 尔 · 杰 克 逊 的 资 产 是 多 少 ？ [SEP]\n",
      "INFO:tensorflow:input_ids: 101 6815 1046 2209 185 3345 1046 6849 1112 6598 772 464 8111 119 8409 783 5401 1039 8020 8170 2399 844 5050 8021 138 122 140 102 6815 1046 2209 185 3345 1046 6849 4638 6598 772 3221 1914 2208 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "相似度为： 0.9995204\n",
      "\n",
      "知识三元组2： 迈克尔·杰克逊 厂牌 钢城唱片 摩城唱片 史诗唱片 遗产唱片 mjj制作\n",
      "相似度为： 0.9938148\n",
      "\n",
      "知识三元组3： 迈克尔·杰克逊 净资产 ▲ 11.78亿美元 （2009年估算）[1]\n",
      "相似度为： 0.9997197\n",
      "\n",
      "答案来自三元组： 迈克尔·杰克逊 净资产 ▲ 11.78亿美元 （2009年估算）[1]\n",
      "相似度为： 0.9997197\n",
      "\n",
      "\u001b[1;31m答案是：▲ 11.78亿美元 （2009年估算）[1]\u001b[0m\n",
      "\n",
      "\u001b[1;31m请提问:\u001b[0m\n",
      "\n",
      "\n",
      "再见啦！\n"
     ]
    }
   ],
   "source": [
    "kbqa()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "至此，使用BERT进行基于知识库的问答系统搭建完毕。\n",
    "\n",
    "本实战营系列的自然语言处理（NLP）领域的四期案例到此也告一段落。\n",
    "\n",
    "NLP是深度学习、人工智能非常重要且复杂的领域，本实战营只是带领大家浅显地了解，有待大家持续探索！"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
